Audio-Visual Emotion Recognition Using Semi-Coupled HMM and Error-Weighted Classifier Combination

This paper presents an approach to automatic recognition of emotional states from audio-visual bimodal signals using semi-coupled hidden Markov model and error weighted classifier combination for Human-Computer Interaction (HCI). The proposed model combines a simplified state-based bimodal alignment strategy and a Bayesian classifier weighting scheme to obtain the optimal solution for audio-visual bimodal fusion. The state-based bimodal alignment strategy is proposed to align the temporal relation of the states between audio and visual streams. The Bayesian classifier weighting scheme is adopted to explore the contributions of different audio-visual feature pairs for emotion recognition. For performance evaluation, audio-visual signals with four emotional states (happy, neutral, angry and sad) were collected. Each of the invited four subjects was asked to utter 10 sentences to generate emotional speech and facial expression for each emotion. Experimental results show the efficiency and effectiveness of the proposed method.